Getting rid of "OK Google": Individual Multimodal Input Adaption in Real World Applications

Multimodal Interaction has the potential to significantly increase the ease of use of human computer interaction (HCI). At the same time, due to error-prone recognition based inputs, it is merely used in real-world applications. While literature on multimodal input fusion describes modeling of different user behaviors as a key for increased robustness, it still failed to prove it’s practical use. This article presents the design of a user study that applies previous theoretical work on individual user adaption in a smartwatch scenario. We describe the practical implementation of a process for error recognition and recovery based on the history of multimodal inputs and a concrete scenario suitable for evaluating its practical impact on the ease of use. This could prove the real-world use of individual multimodal input adaption and finally lead to multimodal systems less cumbersome than today.